Skip to content

fix: run process I/O on a pool that grows to the blocking work - #114

Merged
alfespa17 merged 2 commits into
masterfrom
fix/blocking-io-pool
Nov 18, 2026
Merged

alfespa17 merged 2 commits into
masterfrom
fix/blocking-io-pool

Conversation

@Macbet

@Macbet Macbet commented Nov 18, 2026

Copy link
Copy Markdown

Supersedes #103 by @putz612. These are the same two commits, with authorship kept, rebased onto current master. The fork belongs to an organization, so maintainers can't push to its branch.

The only conflict was at the end of TerraformClientTest: master (#111) added the destroy/-json tests in the same place where #103 added the await overload. The resolution keeps both. Nothing else changed.

Why

TerraformClient runs two blocking stream readers and a blocking waitFor() per command on newWorkStealingPool(). That pool is a ForkJoinPool with one worker per CPU, and it does not grow for plain blocking tasks. When 3N − 1 > P (N commands, P CPUs), some reader never gets a thread, and its child blocks in write() once the pipe buffer fills. This PR switches to a cached pool of daemon threads, so each live process uses exactly three threads.

The most likely real-world trigger is a single command on a 1-CPU executor pod that writes more than 64 KB to stderr, for example with TF_LOG. On master that hangs. Reproduced on JDK 17, 21 and 25.

Verification

All runs in Docker (maven:3.9-eclipse-temurin-17, --init):

  • This branch, full verify: 36/36 in terraform-client, 1/1 in the sample, at default CPUs and with -XX:ActiveProcessorCount=1. Also passed at =2 during review.
  • Master plus the new everyConcurrentCommandDeliversItsOutput: fails. With 5 commands on 11 processors the delivered stdout counts were [20000, 20000, 20000, 20000, 0].
  • Plain master at ActiveProcessorCount=1 already fails ProcessLauncherTest.errorStreamIsFullyCapturedWhenExitCodeFutureCompletes. The second commit fixes that harness.

Notes for the release

  • getExecutor() (Lombok @Getter) now returns a ThreadPoolExecutor at runtime instead of a ForkJoinPool. The declared type is unchanged.
  • Reader threads get TerraformClient's classloader as context classloader instead of the system classloader.

putz612 and others added 2 commits November 18, 2026 03:19
The client's executor was newWorkStealingPool(), a ForkJoinPool sized to
availableProcessors(), but every task submitted to it blocks: two stream
readers and a waitFor() per command. A ForkJoinPool only grows workers
for a ManagedBlocker, which neither readLine() nor waitFor() is, so N
concurrent commands need 3N blocked threads and can never have more than
availableProcessors().

Starvation begins at 3N-1 > P, so on a four-core machine the second
concurrent command already has a reader that is never scheduled. That
command delivers no output at all, and once it has written more than a
pipe buffer with nobody draining it, the child blocks in write() and
stops making progress.

A cached pool grows to the blocking work and reaps idle threads after
60s. It is unbounded in principle, but the bound here is the consumer's:
three threads per live terraform process, which already costs far more
than a thread. Threads are daemon, as ForkJoinPool's workers already
were, so JVM exit behaviour is unchanged.

everyConcurrentCommandDeliversItsOutput sizes itself to the machine,
launching availableProcessors()/3 + 2 commands so that 3N-1 > P holds at
any core count without a JVM flag or a surefire change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test class holds its own static work-stealing pool and constructs
ProcessLauncher against it directly, so the client's fix does not reach
it. It has the same problem for the same reason, and it is why
errorStreamIsFullyCapturedWhenExitCodeFutureCompletes cannot pass at one
core: that test writes 50000 lines to stderr only, and at parallelism 1
the stdout reader takes the single worker while the stderr reader never
runs, so the child blocks on write.

With this the suite passes at ActiveProcessorCount 1, 2 and 4. Without
it, one core still fails on that test, as it does today.

Three lines rather than one because Executors.defaultThreadFactory()
creates non-daemon threads while ForkJoinPool's workers are daemon;
keeping them daemon preserves what this pool already had. Nothing else
in the class depends on work-stealing -
timeoutFiresEvenWhenTheLauncherPoolHasNoSpareWorker builds its own
ForkJoinPool(1) and never touches the shared pool.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@alfespa17
alfespa17 merged commit 60bdd63 into master Nov 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants